Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve file manage to support delete or overwrite files #393

Merged
merged 23 commits into from
Dec 26, 2024

Conversation

SuperEver
Copy link
Contributor

No description provided.

@SuperEver SuperEver marked this pull request as ready for review December 17, 2024 02:15
@SuperEver SuperEver force-pushed the rag/improve_file_manage branch from 3bca1fe to 7bdeed6 Compare December 17, 2024 02:19
zhangyongchao added 7 commits December 19, 2024 13:53
   thread
2. add delete_obsolete_files in doclistmaager, it will find
   status==delting and reference_count==0 file to delete
3. Add reference_count, when adding to kbgroup document reference_count
   will be updated to +1, when deleting from kbgroup it will be updated
   to -1
4. Add KBOperationLogs, currently only deleting file operation will be
   recoreder
5. Modifiy worker thread. when group is default, it will check to delete
   obsolete file
@wzh1994
Copy link
Contributor

wzh1994 commented Dec 24, 2024

修一下冲突吧也

zhangyongchao added 3 commits December 24, 2024 14:17
   and  it'll make database abnormal
2. Modify worker's _worker_reparse_files, it will do reparsing without
   change database
3. Move filepath check to DocManager, DocListManager only provides api
   for finding paths in database
4. Modify DocListManager.get_docs_need_reparse, use one query to get all
   docs.
5. Fix lint problem
zhangyongchao added 3 commits December 25, 2024 19:30
2. As checking and doing is not done in the same time, add KBPathLock to
   lock the paths.
3. For the same reason, reparsing will not support waiting status.
@mergify mergify bot added the lint_pass label Dec 26, 2024
@wzh1994 wzh1994 changed the title Rag/improve file manage improve file manage to support delete or overwrite files Dec 26, 2024
@wzh1994 wzh1994 merged commit f55e91a into LazyAGI:main Dec 26, 2024
15 of 19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants